AI621 A1: Image Processing Pipeline

20215584 Wonjoon Chang

Initials

raw_img = imread('./data/banana_slug.tiff')
raw_img = 2856×4290 uint16 행렬
2217 2391 2229 2456 2223 2481 2241 2474 2220 2448 2258 2521 2259 2483 2257 2505 2199 2463 2212 2432 2241 2468 2199 2501 2207 2449 2201 2466 2227 2471 2195 2439 2199 2424 2216 2390 2185 2358 2191 2326 2170 2337 2174 2299 2167 2315 2158 2305 2163 2251 2407 2209 2467 2237 2440 2237 2501 2250 2428 2250 2482 2236 2486 2285 2545 2272 2447 2285 2462 2249 2483 2287 2472 2274 2494 2248 2483 2217 2506 2248 2444 2223 2452 2197 2395 2196 2332 2183 2346 2167 2306 2149 2286 2160 2306 2142 2311 2188 2285 2158 2239 2437 2244 2481 2235 2438 2240 2439 2262 2481 2239 2491 2264 2485 2242 2493 2221 2477 2242 2489 2247 2527 2218 2509 2230 2439 2219 2467 2252 2471 2236 2490 2216 2340 2193 2389 2197 2348 2200 2338 2165 2319 2175 2290 2174 2284 2161 2274 2141 2284 2409 2223 2453 2210 2426 2197 2439 2230 2429 2236 2479 2247 2516 2296 2511 2251 2462 2260 2493 2248 2547 2263 2491 2299 2498 2259 2513 2266 2469 2241 2499 2253 2474 2190 2424 2220 2383 2189 2362 2173 2379 2159 2310 2177 2293 2140 2309 2146 2314 2142 2197 2412 2204 2442 2225 2478 2261 2481 2265 2528 2233 2521 2235 2549 2238 2475 2250 2527 2212 2494 2269 2503 2240 2490 2250 2558 2253 2475 2224 2476 2227 2466 2204 2459 2222 2412 2212 2374 2181 2337 2175 2351 2184 2312 2163 2260 2158 2283 2145 2245 2399 2194 2427 2220 2484 2220 2486 2214 2470 2254 2540 2275 2483 2262 2502 2295 2528 2260 2468 2235 2473 2275 2516 2264 2497 2324 2496 2267 2510 2269 2498 2204 2470 2222 2479 2199 2410 2202 2336 2160 2371 2166 2329 2158 2276 2158 2306 2149 2257 2137 2203 2480 2215 2432 2276 2472 2275 2479 2225 2504 2226 2483 2254 2551 2229 2496 2234 2509 2247 2515 2266 2538 2215 2464 2230 2533 2269 2551 2218 2490 2224 2513 2231 2449 2244 2418 2221 2424 2193 2403 2155 2330 2153 2328 2173 2287 2175 2265 2168 2239 2374 2204 2430 2199 2462 2207 2465 2211 2471 2285 2494 2246 2459 2236 2460 2237 2437 2228 2509 2229 2504 2248 2504 2254 2571 2271 2519 2258 2491 2225 2500 2237 2457 2222 2392 2214 2438 2203 2383 2132 2398 2169 2363 2149 2361 2146 2270 2126 2323 2126 2227 2461 2241 2467 2251 2477 2228 2494 2255 2539 2218 2530 2227 2476 2235 2476 2224 2459 2249 2555 2265 2507 2265 2547 2272 2536 2212 2552 2276 2512 2240 2448 2216 2433 2259 2457 2207 2399 2213 2380 2203 2417 2197 2347 2159 2303 2153 2294 2132 2285 2415 2182 2400 2195 2444 2233 2478 2219 2464 2218 2458 2215 2524 2217 2506 2239 2469 2213 2496 2264 2536 2268 2515 2269 2534 2224 2527 2214 2500 2203 2465 2216 2431 2168 2437 2169 2409 2209 2418 2187 2381 2182 2403 2149 2323 2187 2310 2164 2271 2148
class(raw_img)
ans = 'uint16'
The type of interger is uint16. Each integer consists of 16bits.
[H, W] = size(raw_img)
H = 2856
W = 4290
Height: 2856, Width: 4290
img = double(raw_img)
img = 2856×4290
2217 2391 2229 2456 2223 2481 2241 2474 2220 2448 2258 2521 2259 2483 2257 2505 2199 2463 2212 2432 2241 2468 2199 2501 2207 2449 2201 2466 2227 2471 2195 2439 2199 2424 2216 2390 2185 2358 2191 2326 2170 2337 2174 2299 2167 2315 2158 2305 2163 2251 2407 2209 2467 2237 2440 2237 2501 2250 2428 2250 2482 2236 2486 2285 2545 2272 2447 2285 2462 2249 2483 2287 2472 2274 2494 2248 2483 2217 2506 2248 2444 2223 2452 2197 2395 2196 2332 2183 2346 2167 2306 2149 2286 2160 2306 2142 2311 2188 2285 2158 2239 2437 2244 2481 2235 2438 2240 2439 2262 2481 2239 2491 2264 2485 2242 2493 2221 2477 2242 2489 2247 2527 2218 2509 2230 2439 2219 2467 2252 2471 2236 2490 2216 2340 2193 2389 2197 2348 2200 2338 2165 2319 2175 2290 2174 2284 2161 2274 2141 2284 2409 2223 2453 2210 2426 2197 2439 2230 2429 2236 2479 2247 2516 2296 2511 2251 2462 2260 2493 2248 2547 2263 2491 2299 2498 2259 2513 2266 2469 2241 2499 2253 2474 2190 2424 2220 2383 2189 2362 2173 2379 2159 2310 2177 2293 2140 2309 2146 2314 2142 2197 2412 2204 2442 2225 2478 2261 2481 2265 2528 2233 2521 2235 2549 2238 2475 2250 2527 2212 2494 2269 2503 2240 2490 2250 2558 2253 2475 2224 2476 2227 2466 2204 2459 2222 2412 2212 2374 2181 2337 2175 2351 2184 2312 2163 2260 2158 2283 2145 2245 2399 2194 2427 2220 2484 2220 2486 2214 2470 2254 2540 2275 2483 2262 2502 2295 2528 2260 2468 2235 2473 2275 2516 2264 2497 2324 2496 2267 2510 2269 2498 2204 2470 2222 2479 2199 2410 2202 2336 2160 2371 2166 2329 2158 2276 2158 2306 2149 2257 2137 2203 2480 2215 2432 2276 2472 2275 2479 2225 2504 2226 2483 2254 2551 2229 2496 2234 2509 2247 2515 2266 2538 2215 2464 2230 2533 2269 2551 2218 2490 2224 2513 2231 2449 2244 2418 2221 2424 2193 2403 2155 2330 2153 2328 2173 2287 2175 2265 2168 2239 2374 2204 2430 2199 2462 2207 2465 2211 2471 2285 2494 2246 2459 2236 2460 2237 2437 2228 2509 2229 2504 2248 2504 2254 2571 2271 2519 2258 2491 2225 2500 2237 2457 2222 2392 2214 2438 2203 2383 2132 2398 2169 2363 2149 2361 2146 2270 2126 2323 2126 2227 2461 2241 2467 2251 2477 2228 2494 2255 2539 2218 2530 2227 2476 2235 2476 2224 2459 2249 2555 2265 2507 2265 2547 2272 2536 2212 2552 2276 2512 2240 2448 2216 2433 2259 2457 2207 2399 2213 2380 2203 2417 2197 2347 2159 2303 2153 2294 2132 2285 2415 2182 2400 2195 2444 2233 2478 2219 2464 2218 2458 2215 2524 2217 2506 2239 2469 2213 2496 2264 2536 2268 2515 2269 2534 2224 2527 2214 2500 2203 2465 2216 2431 2168 2437 2169 2409 2209 2418 2187 2381 2182 2403 2149 2323 2187 2310 2164 2271 2148

Linearization

Identify the maximum value and the minimum value of the 2D-array.
max(img, [], 'all')
ans = 15303
min(img, [], 'all')
ans = 2002
Clip the values within the range [2047, 15000] to remove the effect of dark noise or saturation.
img(img <= 2047) = 2047;
img(img >= 15000) = 15000;
Convert the image into a linear array within the range [0, 1].
img = img-2047;
img = img/(15000-2047)
img = 2856×4290
0.0131 0.0266 0.0141 0.0316 0.0136 0.0335 0.0150 0.0330 0.0134 0.0310 0.0163 0.0366 0.0164 0.0337 0.0162 0.0354 0.0117 0.0321 0.0127 0.0297 0.0150 0.0325 0.0117 0.0350 0.0124 0.0310 0.0119 0.0323 0.0139 0.0327 0.0114 0.0303 0.0117 0.0291 0.0130 0.0265 0.0107 0.0240 0.0111 0.0215 0.0095 0.0224 0.0098 0.0195 0.0093 0.0207 0.0086 0.0199 0.0090 0.0157 0.0278 0.0125 0.0324 0.0147 0.0303 0.0147 0.0350 0.0157 0.0294 0.0157 0.0336 0.0146 0.0339 0.0184 0.0384 0.0174 0.0309 0.0184 0.0320 0.0156 0.0337 0.0185 0.0328 0.0175 0.0345 0.0155 0.0337 0.0131 0.0354 0.0155 0.0306 0.0136 0.0313 0.0116 0.0269 0.0115 0.0220 0.0105 0.0231 0.0093 0.0200 0.0079 0.0185 0.0087 0.0200 0.0073 0.0204 0.0109 0.0184 0.0086 0.0148 0.0301 0.0152 0.0335 0.0145 0.0302 0.0149 0.0303 0.0166 0.0335 0.0148 0.0343 0.0168 0.0338 0.0151 0.0344 0.0134 0.0332 0.0151 0.0341 0.0154 0.0371 0.0132 0.0357 0.0141 0.0303 0.0133 0.0324 0.0158 0.0327 0.0146 0.0342 0.0130 0.0226 0.0113 0.0264 0.0116 0.0232 0.0118 0.0225 0.0091 0.0210 0.0099 0.0188 0.0098 0.0183 0.0088 0.0175 0.0073 0.0183 0.0279 0.0136 0.0313 0.0126 0.0293 0.0116 0.0303 0.0141 0.0295 0.0146 0.0334 0.0154 0.0362 0.0192 0.0358 0.0157 0.0320 0.0164 0.0344 0.0155 0.0386 0.0167 0.0343 0.0195 0.0348 0.0164 0.0360 0.0169 0.0326 0.0150 0.0349 0.0159 0.0330 0.0110 0.0291 0.0134 0.0259 0.0110 0.0243 0.0097 0.0256 0.0086 0.0203 0.0100 0.0190 0.0072 0.0202 0.0076 0.0206 0.0073 0.0116 0.0282 0.0121 0.0305 0.0137 0.0333 0.0165 0.0335 0.0168 0.0371 0.0144 0.0366 0.0145 0.0388 0.0147 0.0330 0.0157 0.0371 0.0127 0.0345 0.0171 0.0352 0.0149 0.0342 0.0157 0.0395 0.0159 0.0330 0.0137 0.0331 0.0139 0.0323 0.0121 0.0318 0.0135 0.0282 0.0127 0.0252 0.0103 0.0224 0.0099 0.0235 0.0106 0.0205 0.0090 0.0164 0.0086 0.0182 0.0076 0.0153 0.0272 0.0113 0.0293 0.0134 0.0337 0.0134 0.0339 0.0129 0.0327 0.0160 0.0381 0.0176 0.0337 0.0166 0.0351 0.0191 0.0371 0.0164 0.0325 0.0145 0.0329 0.0176 0.0362 0.0168 0.0347 0.0214 0.0347 0.0170 0.0357 0.0171 0.0348 0.0121 0.0327 0.0135 0.0334 0.0117 0.0280 0.0120 0.0223 0.0087 0.0250 0.0092 0.0218 0.0086 0.0177 0.0086 0.0200 0.0079 0.0162 0.0069 0.0120 0.0334 0.0130 0.0297 0.0177 0.0328 0.0176 0.0334 0.0137 0.0353 0.0138 0.0337 0.0160 0.0389 0.0141 0.0347 0.0144 0.0357 0.0154 0.0361 0.0169 0.0379 0.0130 0.0322 0.0141 0.0375 0.0171 0.0389 0.0132 0.0342 0.0137 0.0360 0.0142 0.0310 0.0152 0.0286 0.0134 0.0291 0.0113 0.0275 0.0083 0.0218 0.0082 0.0217 0.0097 0.0185 0.0099 0.0168 0.0093 0.0148 0.0252 0.0121 0.0296 0.0117 0.0320 0.0124 0.0323 0.0127 0.0327 0.0184 0.0345 0.0154 0.0318 0.0146 0.0319 0.0147 0.0301 0.0140 0.0357 0.0141 0.0353 0.0155 0.0353 0.0160 0.0405 0.0173 0.0364 0.0163 0.0343 0.0137 0.0350 0.0147 0.0317 0.0135 0.0266 0.0129 0.0302 0.0120 0.0259 0.0066 0.0271 0.0094 0.0244 0.0079 0.0242 0.0076 0.0172 0.0061 0.0213 0.0061 0.0139 0.0320 0.0150 0.0324 0.0157 0.0332 0.0140 0.0345 0.0161 0.0380 0.0132 0.0373 0.0139 0.0331 0.0145 0.0331 0.0137 0.0318 0.0156 0.0392 0.0168 0.0355 0.0168 0.0386 0.0174 0.0378 0.0127 0.0390 0.0177 0.0359 0.0149 0.0310 0.0130 0.0298 0.0164 0.0317 0.0124 0.0272 0.0128 0.0257 0.0120 0.0286 0.0116 0.0232 0.0086 0.0198 0.0082 0.0191 0.0066 0.0184 0.0284 0.0104 0.0273 0.0114 0.0306 0.0144 0.0333 0.0133 0.0322 0.0132 0.0317 0.0130 0.0368 0.0131 0.0354 0.0148 0.0326 0.0128 0.0347 0.0168 0.0378 0.0171 0.0361 0.0171 0.0376 0.0137 0.0371 0.0129 0.0350 0.0120 0.0323 0.0130 0.0296 0.0093 0.0301 0.0094 0.0279 0.0125 0.0286 0.0108 0.0258 0.0104 0.0275 0.0079 0.0213 0.0108 0.0203 0.0090 0.0173 0.0078
Check the intermediate result. Since it is very dark to recognize the process, I multiply the values with 5 for visualization before brightening & gamma correction.
figure;
imshow(min(1, img*5));

Identifying the Correct Bayer Patterns

In the RAW image, each pixel captures only one color. We have to select which version of the Bayer patterns is applied to the image. To construct an RGB image with a size of one-fourth, I choose three values according to the Bayer pattern from each 2x2 square of the image file. It helps to identify the applied Bayer pattern before the interpolation process. In the case of G, I select one value from two candidates.
img1 = img(1:2:end, 1:2:end);
img2 = img(1:2:end, 2:2:end);
img3 = img(2:2:end, 1:2:end);
img_grbg = cat(3, img2, img1, img3);
img1 = img(1:2:end, 1:2:end);
img2 = img(1:2:end, 2:2:end);
img3 = img(2:2:end, 2:2:end);
img_rggb = cat(3, img1, img2, img3);
img1 = img(1:2:end, 1:2:end);
img2 = img(1:2:end, 2:2:end);
img3 = img(2:2:end, 2:2:end);
img_bggr = cat(3, img3, img2, img1);
img1 = img(1:2:end, 1:2:end);
img2 = img(1:2:end, 2:2:end);
img3 = img(2:2:end, 1:2:end);
img_gbrg = cat(3, img3, img1, img1);
 
figure;
subplot(2,2,1);
imshow(min(1,5*img_grbg))
title('grbg');
subplot(2,2,2);
imshow(min(1,5*img_rggb))
title('rggb');
subplot(2,2,3);
imshow(min(1,5*img_bggr))
title('bggr');
subplot(2,2,4);
imshow(min(1,5*img_gbrg))
title('gbrg');
I plot four images with only one-fourth the pixels of the original image. The selected pixels are different according to the version of Bayer patterns. From the results, I think that the version of applied Bayer pattern is 'rggb'. I focus on the lip, the banana, and the shirt. In the case of 'rggb', the color of the lip seems more reddish, the color of the banana is yellow, and the color of the shirt seems blue. It is more natural than the other cases.
figure;
imshow(min(1,5*img_rggb))

White Balancing

Check what the image looks like under both white balancing algorithms. From the previously selected RGB values, I compute the average values and the maximum values. Then, I apply two balancing algorithms.
imgR = img(1:2:end, 1:2:end);
imgG = img(1:2:end, 2:2:end);
imgB = img(2:2:end, 2:2:end);
 
avg_R = mean(imgR, "all");
avg_G = mean(imgG, "all");
avg_B = mean(imgB, "all");
max_R = max(imgR,[],"all");
max_G = max(imgG,[],"all");
max_B = max(imgB,[],"all");
 
img_gwa = cat(3, imgR*avg_G/avg_R, imgG, imgB*avg_G/avg_B);
img_wwa = cat(3, imgR*max_G/max_R, imgG, imgB*max_G/max_B);
 
figure;
subplot(1,2,1);
imshow(min(1,5*img_gwa))
title('Grey world assumption');
subplot(1,2,2);
imshow(min(1,5*img_wwa))
title('White world assumption')
Under the White world assumtion, the image becomes too much reddish. The colors of the skin and the soil are more natural under the Grey world assumption. From the results, I determine to use the Grey world assumption.
figure;
imshow(min(1,5*img_gwa))
imgR = imgR*avg_G/avg_R;
imgB = imgB*avg_G/avg_B;

Demosaicing

According to the 'rggb' Bayer pattern, I choose the RGB color and interpolate the values to restore the original size of the image. By using interp2 function, I construct the interpolation result for each RGB values and then convert them into one RGB image.
[X, Y] = meshgrid(1:2:W, 1:2:H);
[Xq, Yq] = meshgrid(1:W, 1:H);
dm_R = interp2(X, Y, imgR, Xq, Yq);
[X, Y] = meshgrid(1:2:W, 2:2:H);
[Xq, Yq] = meshgrid(1:W, 1:H);
dm_G = interp2(X, Y, imgG, Xq, Yq);
[X, Y] = meshgrid(2:2:W, 2:2:H);
[Xq, Yq] = meshgrid(1:W, 1:H);
dm_B = interp2(X, Y, imgB, Xq, Yq);
 
img_dm = cat(3, dm_R, dm_G, dm_B);
 
figure;
imshow(min(1,5*img_dm))

Brightness Adjustment and Gamma Correction

For brightening, I check the maximum gray value by using rgb2gray function.
max_gray = max(rgb2gray(img_dm), [], 'all')
max_gray = 0.8227
I compare various brightened results with different scaling parameters.
figure;
subplot(3,2,1);
imshow(min(1,img_dm/max_gray))
title('x 1/(max gray)');
subplot(3,2,2);
imshow(min(1,2*img_dm/max_gray))
title('x 2/(max gray)');
subplot(3,2,3);
imshow(min(1,3*img_dm/max_gray))
title('x 3/(max gray)');
subplot(3,2,4);
imshow(min(1,4*img_dm/max_gray))
title('x 4/(max gray)');
subplot(3,2,5);
imshow(min(1,5*img_dm/max_gray))
title('x 5/(max gray)');
subplot(3,2,6);
imshow(min(1,6*img_dm/max_gray))
title('x 6/(max gray)');
From the results, I think that 4/max_gray is an appropriate choice. In the smaller cases, the images are too dark. On the other hand, some parts in the images, such as skin, are saturated in the larger cases.
After that, I apply tone reproduction by computing non-linear transformation. Finally, I get much more natural image with an appropriate brightness and colors.
img_linear = min(1,4*img_dm/max_gray);
img_nonlinear = min(1,4*img_dm/max_gray);
 
tmp1 = img_linear(img_linear <= 0.0031308);
tmp2 = img_linear(img_linear >= 0.0031308);
 
img_nonlinear(img_linear <= 0.0031308) = 12.92*tmp1;
img_nonlinear(img_linear >= 0.0031308) = (1+0.055)*power(tmp2, 1/2.4)-0.055;
 
figure;
imshow(img_nonlinear)

Compression

By using imwrite function, I store the image in .PNG format and in .JPEG format with quality setting 95.
imwrite(img_nonlinear, './A1_banana.png')
imwrite(img_nonlinear, './A1_banana.jpeg', 'Quality', 95)
png: 17.6MB, jpeg with 95: 3.6MB
.JPEG format compresses the file much more efficiently. The compression ratio is 3.6/17.6=0.2045 (20.45%).
As changing the JPEG quality settings, I compare the quality of the compressed image files.
imwrite(img_nonlinear, './A1_banana_50.jpeg', 'Quality', 50)
imwrite(img_nonlinear, './A1_banana_40.jpeg', 'Quality', 40)
imwrite(img_nonlinear, './A1_banana_30.jpeg', 'Quality', 30)
imwrite(img_nonlinear, './A1_banana_20.jpeg', 'Quality', 20)
imwrite(img_nonlinear, './A1_banana_10.jpeg', 'Quality', 10)
 
img95 = imread('./A1_banana.jpeg');
img50 = imread('./A1_banana_50.jpeg');
img40 = imread('./A1_banana_40.jpeg');
img30 = imread('./A1_banana_30.jpeg');
img20 = imread('./A1_banana_20.jpeg');
img10 = imread('./A1_banana_10.jpeg');
 
figure;
subplot(3,2,1);
imshow(img95)
title('95');
subplot(3,2,2);
imshow(img50)
title('50');
subplot(3,2,3);
imshow(img40)
title('40');
subplot(3,2,4);
imshow(img30)
title('30');
subplot(3,2,5);
imshow(img20)
title('20');
subplot(3,2,6);
imshow(img10)
title('10');
I think that quality setting 30 is the lowest setting. When I set the parameter as 10 or 20, some parts, such as skin, have reddish marks. With quality setting 30, the compression ratio is 0.648/17.6=0.0368 (3.68%).